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AUTHOR 



AbS TRACT 



The report describes the development of a replicable package of training 
materials to enable instructional product developers to prepare first- 
draft materials. The objective of the materials was as follows: 

Given a set of specifications for short-term 
instructional sequences, the trainee will be 
able to produce a first-draft of instruction 
that includes the techniques described in the 
materials . 

Five instructional techniques, derived from laboratory research and selected 
through a survey of eminent instructional psychologi s ts * comprised the 
substance of the materials. The techniques were direct practice, knowledge 
of results, prompting, task description and control of inspection behavior. 
The final package required from 12-15 hours to complete and consisted of 
texts, practice discrimination sequences, models of instruction, revision 
and writing simulations, an annotated bibliography and criterion measures . * 
The report describes the procedures employed to develop each of these 
components . 

Eight developmental and field tests were conducted on the materials, using 
subjects drawn from universities, school s n.v d industry. Results of the 
field tests indicated that the materials were replicable in their instruct- 
ional process and effects. addition, a erics of controlled variation 
studies were conducted i ...r . r.r t.«e de ionrent process for the purpose of 
providing input to project decision-making. These studies are reported in 
detail. 









INTRODUCTION 



Two visible and conflict irg alt e mat l ver. exist for solving the raj or 
problem of instruction in the schools. The first set of solutions depends 
upon analysis and application ot scientific 'inclines and procedures. ;nc 
translation of these findings and procedures results in a font of technology, 
and many prominent figures in the field of instructional psychology advocate 
such on approach (Glaser, 1966, Gagne, 1969; Schuts, 1970; Lusadaine, 1964). 
The thrust cf a econd alternative solution to educational problem centers 
not upon the application of systematic procedures but rather on the redefin- 
ition of the proper scope of Instructional endeavor from something measur- 
able and, hence, verifiable to a more subjective, personal activity. Many 
of the proponents of less structured schooling cone to their position 
because of frustration and disillusionment with their instructional 
experiences, and out of such distress springs their pattern for educational 
improvement . Instructional technologi stsj, on the other hand, operate from 
a base of optimism. They assume that eosjt problems can be solved; cost 
phenomena can be explained. The onus is upon the technologist to select or 
to discover the solution. It was out of jsuch optimism that thin project 
emerged . 

One of the few common agreements in educational thought is that change is 
a complex process. A dictum to employ a particular innovation or procedure 
inevitably is Ignored. The history of instructional innovation is a dark 
and checkered one, and rarely have ideas disseminated through teacher 
education institutions, for instance, demonstrated positive impact o^ the 
achievement of the students. 

In the last six years, however, the reliance for the dissemination of 
’better" approaches has shifted fro- teacher training to those agencies 
which develop instructional materials a;\d procedures, sometimes described 
as systems. Such systems are designed to ; ~oduce achievement of particular 
competencies in their ur.vn>. These materials and procedures arc planned to 
be reproducible a:.d to follow* the basic tenets of empirical development. 

The growing success anc acceptance of early materials developed according 
to empirical procedures have encouraged educators to devote more attention 
to additional instructional development endeavors. Unfortunately, this 
interest has not been matched by an expanding force of personnel with 



raining and expertise in 



re cog-* 



the process of instruct Iona I design. *ne 
cm ot the need for suitably trained research and development individual? 
» contributed to tnc funding of the graduate research training program, 
however, the unquestionably localized, if not insular, consequences ot 
university-based training programs precludes the widespread distribution 
of training. At the sane time, the doctoral degree, heretofore the passport 
to the research realm, is being questioned as a prerequisite for effective 
work in instructional development. One solution to the troublesome problem 
of training development personnel involve ?» the application of product 
development techniques to the training mission. Replicable materials vhicr- 
could be successfully used in a variety of sites present likely means for 
ameliorating the training need. This project was developed, in part, to 
test the notion that training in skills relevant to instructional design 
could be me diated effectively, and primarily, by instructional material*. 
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>967) and field testing (Galipot's, 1969). however, materials relevant to 
the production of instruct ional 1 v sound first drafts were less visible, 
iiriggs (iWo) devotes onlv a small portion of his book specifically to 
instruction And suggests that readers refer to books on ‘programming. 

The best such book, In the writer’s opinion, has been written bv Susan 
Markle (>970)* however, the book is frankly a book on programming. While 
much of the content nay be extrapolated to other instructional formats, 
the learner must do surd; translation for himself. A substantial portion of 
the book Is directed to frame editing. Therefore* the perceived deficiency 
in replicable training materials devoted specifically to the development 
of skill in production of first dre.:t Instructional materials, motivated 
the decision to concentrate the project development effort on this specif i 
area. 

0 vc rv lev of Work *’ 1 an 

In n nonobvtous display of naivete* tr.e author was determined to conduct 
this project the ‘right’ way, that is. In accordance with prescript lens 
for instructional development ventures. Thus , the project began with a 
delightful mixture of sel f -consciousness anc smugness , both of which were 
soon to be dissipated. The project was planned to involve four major 
phases: (1) formulation, specification, and Initial prototyping; (.-*) drnfx 
writing and developmental testing, < revision; And (9) denontitrat ion 
testing. The first stage, where the substance of the project war; designed 
to extend from December 1 to March >. Phase two van scheduled for April 
through June: chase three in Julv: phase four was planned for August and 
September with i:*c f.nal report tv •* prepared in October. plain u-u 

schedule should be kept \ r, mind as .» <:*•*, c ription >'• actual operatic:. 



... o staff r»: mb e r n v e r e n 1 ?• o d. t re r 
numbers of decisions made so tb.at 
e : : o rt vou id be aval 1 ah le . 



a*:#- mu are re c 



m r ms. 



t -,e k 1 r. *)u ir <: 
of the develop?' 
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r'*;ari e One ; Fortru iat icn , bpecl t i cat ion an c t rot or vpi n>* 



ViOU-^arch >9 



i.^caufie the proposal entitled the project td a ?*et of >*ro»i»* specific 
at ions, t: e fonrulaclcr. phase of the project vw relatively brief. Al- 
tnou,\i: me inciu.* icn of five ins t ruct Jena 1 technique# had been « t ipulai ci. 1 . 
:.;*e project 'star t needed first to determine wqich technique** would be 
those selected froa the content of the r-accrl^lo. 

* v ur i:u; thin period the following tonka were ac totxp 1 i shed : 



*. V#* conducted a survey of instructional psychologists to anr.e/is 

their preference {or various instruct lonal techniques. 

*. Survey reaulta were tabulated and presented in a paper at the 
Annual Heeting of the American Educational Renearth Association 
in New York. 



3. Tentative select iern oi six 1 nst ruct ionnl techniques was ewide ; 

practice, prompting, knowledge of results, control of inspection 
behavior, task description and advance organlrer. Selection w as 
Kuided by both the results of the survey and the staff’s attempt 
to render the technique# in instructional s-equencea . 

w. Prclininary contact# were nade for deve lorrental and field tent 
fiitca for npring and Burner. 

r >. Kxanples of short-term instructional sequences in a variety of 
ft abject rwittera were developed. These sequences were to be 
nuitablc tor different aged learners and to provide a »ense of 
variety with which instruction r. i?nt be produced. Tne sequences 
vtre planned to be used in dlflcrl -in.;t ion ;rnd revision sections 
of the project. 



Final specification of Che objectives were punctated and are 
presented below: 



C i ven a sec of # pe c i i i cat 1 or; ^ f r vn i ch t o c h oc *♦ e , 
the trainee would be able to develop an ins true t i ora 1 
sequence which exhibited the n: tnr instructional 

techniques included in the r*i t e r i a is . 

subordinate competencies were rcutuiatrd /l*» follow*. 



Ability to d i m c r iri r.at e the .y.f of such : ecr.ni qu»»s In 
sample ina t ruct ional ntrquenreu, pc t: on a single and 

multiple discrimination biwpt . 

Ab i 1 1 t y to rflvifle ftivrr. n eq u e n e e n to i — n r ov e t h e u s e 
of such techniques. 



Ab I i 1 1 y to correctly a e 1 c c ?. « t a t erae n t * re l a t i ve to the 
recommended uae of technique* . 
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of rir.vr. t^nch^r education card it i n r * : <* at a local at ate coil*'* 
btv.ier.tfl wore *.flkrd to di *»c r i - 1 r*ate the t^chniquea in ir.ntruc 
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r *t«jdoat*» and to l<*nd crod^r*or to t.n<‘ rotten of r« i4 *oar :. 
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IXsriny. U:p hinv and e a r 1 v June, tfxtr* , criterion 1 terv? ^nd practice 
cxcrcifltfi vf rr written for the second subticctlon, ’’TrchniGues for 
Reflponac Control. *' The techniques of prompting, direct practice and 
knwlfdgc of result*? forced the aubatance of the component . 

b. ots of spec 1 f 1 c at i or.s were prcrfuct'd for the nlculftt. ion 
the r*A t r ri a 1 . Di re c 1 1 ems for uii e of t he a pc clficnti on?5 
pro vi ded . 

7. In. late June , thirty copiP* of the rvjterial« were produced. 

Work proceeded on the bibliography. 

Chon tacts for field test nite.n were pursued. 



sect ion of 

/ 

were nlso 



hh..*u*e . nrec : Testing and Re vi n i on Cycler? — Jij 1 v-Novrnbe r * 9 J ’ 

•y\ eld teat on Ren poo a *■ Cor t r o i sect i or, va* conducted Julv ,? and- 1 
in Santa Hcmica. 

. Directions and itrnn were revised and rviterini* ^verc tested at 
Arisen* State on undergraduate * . N‘c. staff nrnhe rV^ w< *rc di recti*.* 
involved in the field tear. 

Da: a fro- Santa Men i c a and ArirCr.a were Scored. Revisions were ~Adr 
in. Iterji, scoring procedures, practice di ac rir.i nat i on exercise* and 
text rvu eri ala . 

Yi * -Id trial van conducted cr. Julv 2? and 2 J ir. Torrance. Staff 



rve o , 


hut the train! nr vw 


a\:pervi^ed h • ■ 


local 


: r ttr le.ade rs . 


we re 


c oop lied, an A 1 v x e d ar. 


d fevi« jr.r.r. i; 


s . : *> \ i 


f i t e rj\ 4 practice 



d 1 it c r i r-i n a t i cn exercise we :r :•*. 

*H . •* * t eri «? 1 *> vc rt »* t: V f t • c • : t c e x t e rr. a 1 expert r *■ vi cv b v tv o 
educations! pavcr.oiog i ntv* . Their rvsrcnsrS resulted in rue sirpii- 
f i c a t i on c : 1 an gu a r e errp i c v e d in run t e r i a 1 a an well as rr d i f i c a t l c r. 
ir< the instructional npcuencr. 

revision k-tc rr.hiffl were produced. 



■*V<ieln of instruct ior.a) neoiience* were developed and included i n 
"-.ax e r 1 a 1 * . 



. :-.e pretest van rrdilird :c re tire: revision rath, or than dir. crrir.i: 

a t i or* he h a v leni . 



A field teat vjw conducted at Kar.hac 1 4tr. reach under optical cor.dit. 1 
Data were considered to represent tr*.o first d^rtonr* t rat 1 on trial. 



Da: a ve re anal 



red and 
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•ade . 
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I.!. "retell adr.I r. i s t ered to UCLA teacher education car.d 1 dAt e.s . Field 
trial not pursued. 

i C Individual revision exercises ‘for each Instructional technique' 
were produced. 

‘4. direct ions writ ter, for ‘hands-off* field test At Aritonn htflte. 

* j . Modifications were rjuic for lahoratorv field tests. 

*s. Revisions lr. directions for Pacific Bell Cor. pan v field trial. 

a c i 0 Four: Final Da t a An a 1 vs 1 s , ?jt v 1 «? ! cr. f an d Report P re p a r a 1 1 on 

!>e c e mb e r - Mn r c : . * * 7 1 

’ s L/w lr. the receipt of data de laved product IVr, of final version. 

1 . TV. r b i h 1 i 0 p r a r h. v v as critic*- ;e d a r. d serf c u *i 1 v re vl s e d . 

3.- An optional section to or?»inl;er. was reintroduced. 

i 

3. Introduction to materials wa*. written which incorporated the hl^tcrv 
of development of this prelect. 

:> . The research studies .'one rated through the prelect were compiled. /. 

paper wan prepared nr.d presented at the /Annual Meet in*- of the Are ri car. 
Fducat icrnAl Research Assocl nt i or. in Chicago. 

t- , Fro f easier, al drawings were ‘added to ~atr rials. 

I. All data fro**' all trials v.*re Voted re ream! vs is and praphin/. 

“ . Based cr. data. — ir.rr e r ;.r\ : : at. . ■.*“*. a i c. an ;es were instituted 1 r. 
r ji t e r i a 1 s . 

•. Final cop:.s aerials vr:?* : . ne r at e d . proofed and re r rcuuc.od . 

' • . P re par at icr*. : 11 r.a ■ rep r r : h e ► a r. 1 r Ma r c h. . 
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Tyx * C r i c e r 1 on__I t e 

Information items, which tapped immediate comprehension of the material 
presented in the text segments, underwent an extensive deve lopment ai 
sequence. The tension in the developncnt plan derived from the need to 
have the iters represent the critical areas of substance in the text 
while not requiring 00 many items that the field testing time would be 
enormously expanded. A prominent solution to such problems would invoke 
the use of item sampling techniques, where different 1 tens were distributed 
tc different subjects. This solution was considered but eliminated both 
because of the clerical requirements it would impose on a thin staff and 
the restricted size of our field testing samples. 

items were generated in accordance with item form procedures (see Hively, 
Patterson, Parc, 1968), in that the domains to be covered were identified 
*nd rules for generating incorrect responses were Initially described. 

A standard item format van selected, requiring a cult iple-choice response 
with four options. Each item was inspected following field trials. Ve had 
hoped to organize our field trials into pairs: the first trial was 

Intended to provide data for the upgrading and refining test items; the 
second trial would provide information to be used for improving the 
instructional materials. Data for each item were arraved on a card, with 
nercentage of responses for each alternative displayed. Items with 
.lfficuities of less than .80 were flagged and carefully scrutinized. 

suibie ambiguities in the phrasing cf certain items were identified. 

...re often, however, revision for the text material itself was implied 
by the item data. 

Placement of criterion items for text material v ns also varied. On eariv 
trials, items relevant to a major area, for instance. Techniques of 
Response Control, were presented together. Or. subsequent trials, however, 
toe text criterion items were placed at the conclusion of the text 
materials to which they were relevant. Text criterion items should not 
be confused with the embedded information items which were inserted in 
the final version of the rr-ate rials. The itemr; interpolated In the* text 
were to serve an Instructional, rather than an evaluation function. 

information it era for use on the pretest ar . d on the retention test were 
randomly selected from the pool of text criterion questions which had 
been identified for all text sections. 



Retention Test 



Fifteen multiple-choice items, three each randomly selected from each of 
the questions relevant to the five text segments, comprised the retention 
test. The purpose of this test was to determine hew much information was 
remembered at the conclusion of the entire training package. The completion 
of the text criterion items described above usually occurred within two or 
three hours of the onset of instruction. The retention test was administered 
at the close of the instructional sequence, from five to twenty hours after 
instruction c omm enced. Revision procedures for these items were identical to 
those described above, and especially poor performance (difficulty below .7), 
on the retention item was noted and revision of item or test was usually 
required . 

Revisi c n Criterion Task 

T vo sample instructional sequences were developed to serve as the bases 
for the revision task, where the learner would be required to demonstrate 
ability to correct instructional sequences with regard to deficiencies in 
the use of particular instructional techniques. Revision exercises were 
scored in three ways, one of which survived to the final trial. The first 
procedure required that the sequence be analyzed in terms of its peculiar 
deficiencies. For instance, if the technique of knowledge of results was 
missing, how many opportunities did the learner have to insert the use of 
the technique. The learner’s score was a proportion of' the number of times 
the technique was used over the number of opportunities available fer use. 
Obviously, this procedure was appropriate for knowledge of results, but 
became more tenuous for the other Instructional techniques. For example, 
a task description, if adequately stated, may need to be included only 
once. Since frequency rules for the use of text-embedded questions, 
practice and prompts were eschewed in favor of empirical trials, the 
arbitrary delineation of numbers in order to derive a proportion did not 
seem to be worth retaining. 

A second procedure involved the assignment of effectiveness ratings from 
five to one by the scorers of the revision exercises. Effectiveness 
parameters were explicated for the scorers and revised so that agreement 
would hopefully be forthcoming. However, the necessary congruence among 
scorers on the assignment of effectiveness scores was never obtained. The 
last procedure employed, and ultimately retained was an on-off judgment 
of the use of each instructional technique. Agreement on the use of tech- 
niques was excellent, essentially perfect. 
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Writing Simulation 



The writing simulation performance test was designed to measure adequately 
the objective of the project. While recognizing that the testing condition 
would be contrived in that it was both time and location controlled, the 
attempt was made to provide relatively realistic stimuli for the writing 
simulation task. The trainee thus was presented with a series of short-term 
instructional objectives suitable for the design of replicable instructional 
material* The learner was directed to employ* where possible, the instruct- 
ional techniques advocated in the materials which they had completed. Sets of 
specifications were developed which included objectives for all grade ranges 
for a variety of subject areas. The criterion task would require the partici- 
pant to select a subject field in his or her area of interest or expertise 
and to spend about one hour preparing a first draft of materials which 
exhibited the appropriate instructional techniques. 

A major problem with a constructed response task as described above is 
deciding on an adequate scoring procedure. A series of scoring alternatives 
were developed and tried on the criterion data obtained from the field 
trials. An early version of the scoring sheet is presented in the Appendix. 
Rather complex decision rules were provided to guide the scorers* use of 
the guideline. 

Responses from twenty-six subjects were scored by two scorers using the 
form above. Pearson product moment correlations were computed by technique. 

TABLE 1 



Correlation Between Two Scorers of Constructed Responses 
By Instructional Techniques 



Technique 



r 



Task Description 
Inspection Behavior 
Direct Practice 
Knowledge of Results 
Prompting 
/ 



.84 

not used 
.49 
.74 
.28 



i 



\ 



Analysis of the scorers* problems through discussion revealed that the 
ambiguities occurred in assigning ’’effectiveness" points rather than in 
determining the presence or absence of the techniques themselves. The form* 
as generated, resulted In a subscore by technique which incorporated both 
"use’ dimension and "effectiveness." A variant of this scoring procedure 
was adopted, where the scorer rated the presence or absence of each tech- 
nique separately. In addition, an effectiveness rating, from 1 to 5 , based 
on a set of explicit criteria was assigned to the use of each technique. 
While scores assigned by independent raters on the presence or absence of 
a technique were virtually Identical, there was considerable variation in 
the correlations obtained between pairs of scorers. Because of the desire 
to have both the instructional and evaluation components of the materials 
approach standards of replicability, the final scoring scheme relied on the 
on-off detection of the presence of a given technique, with an optional 
effectiveness rating system. A copy of the form is included in the Appendix. 
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Attitude Measure 



A measure of interest and attitude toward the materials was developed 
for use In formative evaluation. Questions regarding the length and 
complexity of the materials, clarity of explanation, and usefulness of 
ideas were designed to provide feedback to the development staff for 
future revisions. A copy of the program questionnaire is included in the 
Appendix. 

i 

In addition, a revision sheet was provided which consisted of four 
sections: (1) a space for the trainee to list specific suggestions 

regarding how the material should be revised; (2) a summary impression of 
the materials; (3) a summary statement of how interesting the materials 
were; (4) a personal evaluation of learning. Data from the revision sheet 
were used both as a basis for materials improvement as well as dependent 
measures for controlled variation work to be described in a subsequent 
section of this report. 
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INSTRUCTIONAL COMPONENT DEVELOPMENT 



While the resume of work highlights the project activities chronologically, 
the explicit activities of the project need also to be described in more 
segmented terms. The following section will describe the tasks which were 
required for completion of the project and include elements of procedures 
which might prove useful to prospective instructional development person- 
nel. Where relevant, examples of forms used are included in the Appendix. 



Text 



Once the topic for instruction had been selected, the development of the 
text segments proceeded along fairly conventional lines. The text sections 
were written in an attempt to convey accurate information regarding the 
utility and application of the various techniques. The purpose of each 
text section was to persuade as well as to inform the user regarding the 
value of employing the technique in instruction. The text length was 
limited by the number of discrete information points required by the given 
techniques, resulting in substantial length variations among techniques. 

For instance, the Information statement for the technique of task descrip- 
tion was considerably shorter and based on less theoretical complexity 
than the statement related to the control of inspection behavior. Among 
the rules to guide the production of the text section were the following: 

1. Attempt to extrapolate accurately from referenced research. 

2. Attempt to avoid oversimplification* 

3. Avoid the use of citations and journal-style justifications 
in the text. 

In addition, references to live teaching as well as instructional materials 
development projects were Included, primarily to assuage the teacher educa- 
tion subjects employed in field tests. 

During the major rewrite of the project (August, 1971), the language was 
simplified and each text section was reorganized to include the following 
categories : 

1. Introduction: designed to motivate the reader. This section 

normally referred to prior school experiences 
of the trainee as a basis for understanding 
the value of the proposed technique. 
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2. Definition: the operational definition of the technique 

employed. 

3. Forms of the technique: descriptions of alternative renderings 

of the technique. 

4. Types of tasks: if possible, the variety of tasks for which 

the technique was particularly suited. 

5. Effective use of technique: frequencies and contexts in which 

the technique optimally might be 
employed. 

Special attention to the theoretical basis for the technique and to the 
types of learners for which the technique might be particularly appropriate 
were also included. During a special field trial, a comparison between the 
use of summaries and text-embedded adjunct questions was conducted and is 
described in detail in a later section. The final version of text materials, 
however, reflected both treatments: questions were inserted at relevant 

points in the text and brief summaries of the major points of the text 
material were added. 



Practice Discrimination Exercises 

Practice exercises referred to discrimination opportunities for the trainee 
where, within a given segment of an instructional sequence, the presence or 
absence of single or multiple use of the technique was to be determined by 
the learner. 

The discrimination activity was assumed to be prerequisite to the ability to 
revise instructional sequences deficient in the use of given techniques and 
to the ability to generate instruction which Included the specified tech- 
niques. An attempt was made to introduce variations into the instructional 
sequences used in the practice discrimination exercises, so that different 
aged learners, different types of tasks, and different instructional formats 
would be included. This decision was reached after a review was conducted 
on the initial practice exercises designed. These sequences depended 
heavily on cognitive classification tasks and were presented in a very 
linear '’programmed instruction" format, which could easily promote mis- 
modeling by the trainees. 

Specifications to guide the development of the practice discrimination 
exercises were produced and revised. Samples of the single and multiple 
discrimination item forms are included in the Appendix. 




As evidenced in the sample item fortes, the original response required 
of the trainee was a simple yes or no discrimination regarding the use 
of the technique or techniques under scrutiny. It became apparent that 
learners might be able to respond satisfactorily to the exercise by 
merely skimming the sequence. The evil in such practice was suspected 
to lurk in the learner’s reduced exposure to the range and nuance of 
use of the instructional techniques and the reduction in the modeling 
function these exercises might serve. To remedy the situation, learners 
were also asked to rate the "effectiveness" of the use of the technique, 
using for their criteria the recommendations from the text section. As 
knowledge of results sections were provided for the discrimination task, 
a quasi-knowledge of results for this rating behavior was devised. The 
project staff members were asked to rate the use of the technique. Their 
modal response provided the "expert" rating which was included in the 
knowledge of results for the practice exercises. Explanations for low 
ratings were provided as feedback to the trainees. 

A major difficulty with the design of practice exercises was the in- 
ordinate time required of the learner to read through the sequences in 
order to respond to the exercises. Three exercises were designed for 
each instructional technique as well as two comprehensive practice 
exercises which combined the techniques of direct practice, knowledge of 
results and prompting and two exercises which combined task description 
and control of inspection behavior. The decision was made, when reading 
times and fatigue indicators soared on initial trials, to permit the 
trainee to choose to view these exercises as remedial. Another option was 
that the trainee complete two of the three exercises for each technique , 
and if correct on both, skip the third exercise in the set. 



I nstructional Models 

xn our early formulation plan the practice discrimination sequences were 
to serve nultiple purposes. First, they were to provide opportunity for 
the participant to identify the presence or absence of a particular 
technique in a simulated instructional sequence. In the case of positive 
instances, they were also to serve a model function, by demonstrating 
what the use of the technique would ' look like.*' However, during the 
course of the tryouts, suggestions from subjects often raised the need 
for a model or sample of reasonable use of the techniques. To this end, 
a sez of models was prepared and inserted following the practice dis- 
crir,^ nation exercises. These models were to employ the techniques in 
c^ear situations, so that the learners could refer to them as concrete 
samples of the abstractions in the set. An added feature of the models 
was a written supplement, which would explain what was happening in the 
sequence, e.g., "the technique of practice was employed when the problem 
situation was posed." These explanations appeared juxtaposed to the 
instructional sequence at the precise location where the use of the 
technique occurred. Reference to the models was designed to be optional 
and appropriate for those learners who needed more examples than those 
furnished by the practice discrimination exercises. 
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Revision Exercises 



\ 



The ability to locate and remediate specific deficiencies in instruction 
was thought to be prerequisite to the behaviors required in the writing 
of first draft instruction which includes the use of particular instruct- 
ional techniques . For our initial revision exercises, two instructional 
sequences were produced. Each included incorrect use or sometimes the 
absence of use of all five instructional techniques. Learners were asked 
to review each sequence and its specifications and to improve the r.ateriai 
according to the recommendations provided in the text sections. A series 
of difficulties in the design and implementation of these exercises was 
uncovered. First, when the provided instructional sequence was poorlv 
written across the board, or when it represented a particularly trite 
approach to the teaching of the skill, the trainees attempted to rewrite 
the entire exercise. The time they spent in the reformulation of format 
interferred with their correction of the instruction along the guidelines 
expected by the exercise. A second problem related to the nature of the 
task. When directed to correct or improve a sequence according to given 
instructional techniques, some of the participants tended to write notes, 
such as "I’d add more practice*’ or "I’d tell them vhat the objective was” 
without demonstrating that they could do an adequate Job of the task. While 
the identification of a deficiency is clearly prerequisite to the design of 
a remedy, it is not sufficient. The directions provided the trainees were 
improved and structured so that identifying locations of deficiencies and 
the specific remedies proposed were required of the respondent. 

Suggestions from participants following the field tests indicated that 
the transition from multiple discrimination to multiple correction 
too abrupt. The sequence was modified so that five additional revision 
exercises were inserted following the multiple discrimination tasks and 
prior to the comprehensive revision exercises. One exercise for each 
instructional technique was produced, so that the learner needed to fccu s 
only on the use of task description, for example, in his or her initial 
attempt at correcting deficiencies. These sequences exhibited a stronger 
series of prompts, in that the learner's attention was directed only to 
a single technique and that it was clear that there were errors in its 
application in the instructional sequence. Tne comprehensive revision 
exercises required that the learner synthesize two skills: first, multiple 

discrimination, to determine which, if any, techniques were employed in- 
appropriately or not at all; and second, to make the correct applications 
of the technique. 

Confirmation of these exercises was provided by a series of alternative 
models, regarding what modifications were deemed appropriate by the staff. 
Paper exchange with a peer was also encouraged. 



Writing Simulations 

The development of the writing simulations closely adhered to the pro- 
cedures used in preparing the criterion testing situation on first-draft 
production. Sets of specifications were produced which exhibited content 
appropriate to various subject matters and grade levels. 
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An important dimension of the simulation excrciBC was providing a ran;v 
of specif icat Iona from which the learners could choose. We did not vi ,h to 
fail in our task because we had identified only a marrow band of subject 
natters and age levels for the first-draft writing practice. We also 
wished to avoid the freest situation, where the learner would generate 
the specifications for his or her own work. Previous training experience 
Indicated that the specifications which were generated by the learner 
would suffer with high probability from a nurher of def 1 cicncica . The 
specif icat iopa roight not be operational or stated in suf f lc lent dc tai i . 
Thev might be such that thev required only simple responses from the 
intended learners and thus not provide a task of sufficient complexity 
so that the instructional techniques would have the opportunity to be 
displayed. Short of attempting to pet the trainees* specification-writing 
behavior under the program’s control, a task which would deplete our 
staff's resources f the provision of a wide range of alternative specific- 
ations from which the trainee could select his practice situations was 
accepted as a reasonably appropriate stimulus condition. Further ration- 
alization for the process was derived fron previous product development 
work, where the writer, of instruction often has only minimal Input into 
the design of instructional specifications and more often has these 
specifications provided by the management level of the project. 

A number of sets of directions wn 9 written and tried out with successive 
field test groups. In addition, the amount of tire allocated for the 
practice in product writing was varied. In early trials, only a single 
half hour was permitted, usually because of tirx» constraints. In later 
versions, the amount of time was expanded to ov^er an hour. The final 
trials permitted 45 minutes to one hour for each product writing and 
encouraged two attempts by the participants. 

The confirmation of such practice opportur. I : i e • posed a few problems. 

bvieusiy, since the responses were to be tons true ted bv the part icipar. tn , 
some notion of the adequacy of their attempts needed to be communicated. 
The provision of a range of models was me solution which was considered, 
relieving the writing, a series .if sample sequences relating to each 
specification would be presented. The participant would compare the wch 
he or she produced with the models presented and determine whether the 
te_chniques were incorporated in them. This solution va k rejected or. cost- 
effectiveness grounds. On the one '-and, it was feared that the provision 
of samples would unnecessar 1 ly limit the trainees’ view of proper use cf 
t.^e instructional techniques; on the other hand, the amount of staff 
i'.m< required to develop adecuate samples for each of the specifications 
v .icr, might be selected would place excessive demands on our capability. 

. .stead the procedure of peer confirmation was employed. Trainees were 
instructed to work in pairs with another participant. Following the 
designated writing period, a ten-minute exchange and critique period was 
provided, where the trainees would read each other’s efforts and criticize 
the use of the techniques, •-'here questions arose, the trainees referred 
to the text materials provided. Observation of this peer confirmation 
practice revealed no major difficulties. People were evidently able to 
exchange and criticize each other’s work profitably. 

U . 
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Anno: at cd P. ib Uo^r aphy 

the development of the annotated hih liographv was undertaken because It 
wns believed that such a component would contrlbut »• aly.nl f leant ly to 
differentiation of the product’s utility for various groups. Graduate 
students, for instance, would be especially Interested in pursuing sc~« 
of the allegations made In the text sections regard Ins the various tech- 
niques, although It appeared unlikely that school personnel would be 
interested, in general, in the research design, sarpllnR techniques ar/e 
detail of experimental procedure. Pnferences relatinp, to the Riven 
nlques were collected by research assistants. The studies were then re- 
viewed to eliminate apparent overlap or Irrelevant experiments. Abstract 
of the research studies were prepared which Included the following 
categories : problem or hypothesis, subjects, sarpllnp. procedure. (*.'•? , 

treatment length, results, ar.d conclusions. Although the actual nbstrs : 
p rcrvldcd the information with certain variabllltv In description, 
overall purpose was to whet the reader’s appetite for ncre such experi- 
mental writing. The component, because It wan rot essentia, tc the 
accomplishment of the project objective*, was deferred and was cor pie ted 
later ir. the project’s lifespan. Wile net formal lv tested, the Mb,:- 



osrraphv Has ‘undergone two careful reviews 



total rrwr 1 te . 



of local interest ar.d demand at * Cl.A . the h lb lircr aphv is ant icipated 
s valuable addition to programs or. Instructional research ar.d develop' 



The total package including ail the cr-porents require.: arr 
fifteen hours of Instruction and testing to •:c~niete • 
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trials ve r c to vr r : f y 



- ; the product try Cut ft vaTied vitl 
development 'icquenct 1 where they occurred. burl* 
the aultablllty of prototype fomatn <and criterion items. Later trial!* 
served to identify papa In the «ec;uence or.d to provide data for revision 
ci previously developed material ft . 71 w: trial* ala« were deal stood r .o that 
nropreftft tvelv lesa staff involvement in the trveut waft required, in 
-rder ro ascertain whether any desrree c* cr^rar repi i c ab 1 1 1 1 v had t err. 
ft c h 1 e vr d . 
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st totally hv deaisen bur partly bv happens t ance , the subject population 
varied by trial. The ataff believed that the materials should be reared 
rr. different fcinda of people with viric^ft Job re^ui rerwnt ft which ;vr rtalne •: 
to instructional development . Subject* participate? in the field trial* 
represented four distinct pious* prcaervice teacher education candidate* 

. raduate atudcntfli in instructional deve 1 or men: , experienced school 
personnel, and inn t rue t lor. al development ft raff re rf erring in outside 
ipencicn. Tollcpeft and universities provided the source of student** a: 

\ _La (N-29) and teacher education candidates at f-ar. Fernando Valiev r-ta:** 
•.'oiiege The first prototype trials of the directing attention 

comment woa cried on tvn preupn of IX LA teacher education candidate** 
?>*&>. 7. t e prototype mater: 

experienced elementary level school left.’-* r =. The fitftt of i *• 1 :<* 
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Subject F'rocuroriir,; 

A terpen to obtain *i«3b 5«:*c t n occu pied a subt/t<int ial portion o: n t «if f 
ac tent ion throughout the grant per iod ? A fiver developed which 
attempted -to rial tele participants fro--: various school districts (ace 
Appendix), Such requiuata were often transferred from person to person 
within the ntructu.rv or a school district and rarely resulted in a field 
test nice. Arronp. the salient injuries sustained in thin regard wan an 
at leapt to uim: UCLA teacher education studentn in a validation trial 
October «nd ttovcBbcr* *9?t< Their education courses were being conducted 
on a highly individual ited basis. The students rebelled when oitked to. 
complete a pretest in a group aituatiem and anxiety and negative affect 
on the part of the potential subjects and the staff eicnbcra were at ti 
level., where it vaa felt mot productive to abort the field teat, 

A oecond area of din appointment wan the inability to secure participate 
f too regional laboratory ataf f < Ki^ueatn were sent to nix regional 
laboratories and positive response v ere received frorr four* Packets and 
directions were prepared and called, hut no further word van received 
troa three of the sites . The fourth, , the Far West Regional Laboratory, 
had intended that the mat crisis he tried on part Iclponia- in a joint 
project with 'he American institutes for Research, however* 1 Ini ted 
instruct ion precluded ihis^uite of the material a and the packets were 
returned unopened. Explanations for these raapotutea are many; perhaps 
the ' cutter iala were seen to be irrelevant; perhaps the aiaaiem oriental In 
of the development agency interferred with the ability to free staff for 
the approximately fifteen hours required for program training- Flans are 
underway to query the regional laboratory staff members to determine the 
reasons jor the lack of coupler ior- or the mater lain « 

A final point should he cade regarding- the acquisition of Subjects a/of A 
field tenting sites. Sr; all but on** «: the field teats conducted, c!* f e 
contact was initiated by the project director or her coordinator and 
directed to aoocont* with whom previous successful contact had been -radr 
Tn e per cen cage o f po » i t i ve r v suit s o £ con tacts wi t h new people i n 
laboratories, school districts, and other ina t :i tut ions won dep reusing.: v 
x ov\ 



Teat i S che du 1 e 

The program was tested in jiarsponeeAu during the various product tryouts 
The following table profteifts the data, location, number of subjects a;v.; : 
components tea ted at each tryout. 



TABLE 



Tenting. Schedule 




IX'it o 


Locftt-lon 




Corsponentfi Tested 


February 


UCLA „ ■ 


29 


Pretest 


Vti b n; ary 


Son Fernando Valley 
State College 


i i 


I* re ten t 


Apr i 1 


UCLA 


10 


Advance orgnniicr, inspection 
behavior text and discrininat S on 
exercises 


June 


COLA 


18 


Entire component of directing 
attention including texts, Items 
and diacrininatlon exercises 


July 


Santa Houle* 


9 


Response control text, items and 
discrimination exercises and 
writing simulation prototype 


July 


Arizona State 
Calve ni t ty 


18 


Entire package to date: lncludin 
text Items and discrininat ion 
exercises for both response com 
and directing attention, writing 
n iculot ions 


July 


Torrance 


7 


Response control package plus 
revision exercises 


Sept c zh£T 


Konhut tan {loach 


21 


Entire package, including, texts, 
items discrimination exercises, 
models, revision exercises and 
writing simulations 


Novtrijc r 


Arison* State 
Cal ve r a 1 1 y 


i •> 
» 


Sane components an above 


iK‘Cc-^f! r 


Mult ip le Call fora la 
Locations 


H 


Entire package 7 ' 



In Addition, a controlled variation tent w«».s made during the Winter, 59 72 
the use of embedded questions (N*22) . The text evsterlala and items were 
tnc only port ion o of tho programs employed. A supplementary teat of part of 
the aateria-la was taade during January during an experiment coesp 4 ring media v 

variation# of the discrimination exercises (N>50> . Thin otudy will be >* 

reported in * aaparate section. 
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REVISION ACTIVITY 



The b uneti i or revision were many . Thev included data from the tryouts 
of the program on appropriate subjects, reviews by individuals external 
to the project staff and judgments made hv the staff members* Because 
ten separate tryouto had been planned (although only eight mat er ial ized) 
the revision pattern was intended to be controlled by the sequential 
< acquisition of data. Ideally, each version of the propram would be 
revised tux a consequence of judgments made based on datr acquired from 
the moat recent field trial* Thus, data from Trial One should contribute 
to the development of Version Tvo'and no on. Unfortunately* procedural 
problems conspired to upset such progressive application of performance 
data. Field trials, for instance, needed to be scheduled well in advance, 
so organizations could corsni t subjects for the length of time estimated, 
Therefore, the development .team was always working against a schedule: 
the dates that subjects would be ready to begin the propram* Second, many 
of the trials were planned off-site, so that the staff was dependent upon 
the program administrator to mall the raw data hack to them* Often, there 
were inopportune delays* The multiple responsibilities of the staff 
taenbere also compounded the problem, for the project could not afford a 
single peroon with only a data analysis task . .Data analysis responsibili- 
ties nay have been perceived as interrupting core continuous and interest- 
ing development or ’'creative’ activities. 

Thus, the inevitable data analysis lap from the previous tryout, when 
combined with the preordained tryout schedule and the enormous time re- 
quired to prepare multiple copies of materials resulted in a situation 
where data from one field trial were being tabulated at the same tine 
the materials for the next field trial were being prepared. Rather than 
on the next proximate field test, data-basec revisions tended to be in- 
corporated In materials at a later tin*. Thus , data from Trial Four had 
its impact on Six. 

A revision strategy wa*. employed where we could take account of inform- 
ation from the most recent tryouts without unduly deferring material 
preparation and multiple copy production. Data from the immediately pre- 
ceding tryout were employed In two wavs. First, criterion items were 
revised. Item editing and revisions in scoring procedures were possible 
without delaying costly production efforts. Instructions to the user, 
usually contained in brief letter format . were also revised, as were 
directions given to the program odmir.i strator at the field teat site. 

The major revision ‘act ivi cy* occurred i:. August and September, 1971, During 
this period, a rewrite of the entire set of text materials was completed, 
based, 1A part, bn the accumulated data from Trials One through Five and 
the detailed review submitted by Dr. Howard Sullivan of Arizona State 
University* Subsequent modifications to the project from Trials Six 
through tight included variation in sequence and directions to the user, 
addition of specifications for the simulation tasks* inclusion of the 
bibliography and technical modif icati ons in visual design'. 
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The summary of data-based revisions is presented below. 



Version One April, 197 0 

Text Instruction: 

1. Final selection of instructional techniques. 

2. Generation of Item Forms. 

3. Preparation of expository instruction for selective attention 
component . 

Criterion Items : 

Preparation of information items. 

Practice Exercises: 

T. Identification of practice exercises suitable for selective 
n attention. 

2. Adaption of practice exercises to: 

a. Reflect the entire scope of the objective. 

b. Employ positive or negative instances of selective 
attention techniques. 

c. Stylistic improvements. 



/ 
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V ersion Two i June, 1970 
Text Instruction: 

1. Preparation of instruction on Task Description. 

2. Clarification of Directing Inspection Behavior and Advance 
Organizer as indicated by trial data on criterion items and 
practice exercises. 

3. Stylistic editing. 

Criterion Items: 

1. Revision of items as indicated by empirical data. 

2. Addition of items for Directing Inspection Behavior. 



/ 
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Practice Exercises: 



1. 


Graphic cues and prompts added to Practice Exercises on which low 
performance occurred. 


2. 


Directions for responding to Practice Exercises clarified. 


3. 


Elimination of nonessential instruction in Practice Exercises. 



Version Three July, 1970 
Text Instruction: 



i. 


Revision of all instruction in Techniques for Directing Attention 
to eliminate the majority of theoretical rationales and technical 
editing. 

\ 


2. 


Stylistic editing. 


3. 


Shortening of Control of Inspection Behavior as indicated from 
program questionnaire data. 


A. 


Preparation of text instruction for Component II — response control, 
Including practice, prompting, knowledge of results. 



Criterion Items: 



i. 


Revision of Directing Attention items as indicated by data. 


2. 


Preparation of additional items for Advance Organizer and Task 
Description. 


3. 


Preparation of items for Response Control instructional segments. 



Practice Exercises: 



i. 


Replacement of three Techniques for Directing Attention practice" 
exercises whose programmed format and/or subject matter complexity 
rendered them inappropriate for attention direction techniques. 


2. 


Revision of remaining practice exercises as indicated by trial data. 


3. 


Preparation of Practice Exercises for Techniques for Response 
Control. : 



Simulations: 

Elementary level specifications for writing simulations generated. 

21 . 



Version Four July, 1970 
Text ins truct ion : 

No change . 

Criterion Text Items: 

1. Posttest (retention) constructed, 

2. Revised for Response Control segment. 

Practice Exercises: 

No change. 

Simulations: 

Number of specifications expanded. 

User Directions: 

Orientation for leader (nonproject staff member) of session prepared. 

Version Five September, 1970 
Text Instruction: 

1. Complete rewrite, simplifying language of text materials, 

2. Advance Organizer deleted, 

3. Revisions where test performance below ,80. 

Text Criterion Items: ^ 

Rewritten. 

Practice Exercises: 

t 

Edited. 

Simulations: 

Specifications expanded. > 

Revision Practice Sequences : 

Added on Posttest. 
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Model Instructional Sequence Prepared: 

1. Directions for supervisors revised. 

2. User letter prepared. 

•>» * 

Version Six October, 1971 

1. Materials resequenced to be ordered by technique, e.g,, direct 
practice, rather than mode, e.g,, text, 

2. User letter revised 
Supervisor instruction revised 

3. Practice exercises rewritten where potential copyright infringement 
existed. 

4. Revision exercises relevant to each technique added. 

5. Writing Simulations: 

Secondary level specifications added to practice and posttest 
simulations, 

/ 

Version Seven Final Version January, 1972 

Introduction rewritten 

“Organizer 11 section revised and added as an option 
Bibliography added 
Directions to user revised 
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DATA ANALYSIS AND RESULTS 



Data from each tryout are presented in sequential tables below. Tables 
reflect performance on instructional exercises as well as criterion tasks. 

The affective data are reported in two summaries with percentages for 
both positive and negative items presented. 

In reviewing the progress of the project, it is well to keep in mind that 
the subjects varied greatly in experience and predisposition from tryout 
to tryout. Secondly, each field test generally reflects an increasing time 
and content load on the participants. While performance cannot be directly 
compared from trial to trial, because of test item revisions as well as 
sample variation, a sense of the progress of the project can be obtained 
by following the empircal history of the project. 

Technical features of the tables include the use of percentages as the 
most comprehensible transformation, the summary by technique for both 
instruction and criterion, to determine the differential difficulty subjects 
experienced with the material. The number of subjects for summary data varies 
from that described as the^number participating in the tryout because only 
information from those individuals with no missing data were used in the 
analysis. Table 4 presents pretest data while the following tables display 
results from -tryouts . 
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